In Appendix E, we detail our experimental settings and exhibit additional experimental results.

A.2 DerivationsforSection3.1 We begin with a formal derivation of the formulas in Section 3.1. We remind that we consider a function F(θ) whose parameters can be split inton SI groups: θ = (θ1,...,θn). We solve an optimization problem(1)with projected gradient descent(2). Remark2 The above formulation allegedly lacks the third (divergent) regime. If, conversely, η > 1Pn i=1αi, then at each iteration at least one of the individual ELRs exceeds its convergencethreshold: ηi > 1αi.

For the Spectral Mixture Kernel, we use 4 mixtures. The CNF component for our model was inspired by FFJORD. For NGGP, we use the same CNF component architecture as in for the sines dataset. Adding noise allows for better performance when learning with the CNF component. We also use the same CNF component architecture as in the sines dataset. For this dataset, we tested NGGP and DKT models with RBF and Spectral kernels only.

In this paper, we discuss the expressivity and trainability of narrow neural networks. Appendix H introduces the following contents.

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves robustness in the over-parameterized setting. The effect of depth closely depends on the initialization and the training mode. In particular, when initialized with LeCun initialization, depth helps robustness with the lazy training regime. In contrast, when initialized with Neural Tangent Kernel (NTK) and He-initialization, depth hurts the robustness. Moreover, under the non-lazy training regime, we demonstrate how the width of a two-layer ReLU network benefits robustness. Our theoretical developments improve the results by [Huang et al.